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ABSTRACT 



This paper presents results of a review of mail survey 
response literature and provides recommendations for the application of some 
response- enhancing techniques to the Schools and Staffing Survey (SASS) of 
the National Center for Education Statistics. The SASS is an integrated set 
of surveys sponsored by the NCES and conducted by the Bureau of the Census to 
provide data on teachers, students, administrators, librarians, and libraries 
in public and private schools. The SASS uses mail questionnaires as its 
primary method of data collection, but in the most recent survey, only 
between 25% and 75% of surveys have been returned. It would not have been 
wise to produce SASS estimates based on the responses from mail surveys 
alone. After telephone interviews and a few face-to-face interviews, the 
response rate rose to between 80% and 100%, but obviously costs would be much 
reduced with better mail responses. A review of the literature indicates that 
response rate enhancement techniques have been grouped into the following 
general categories: (1) motivating a response; (2) content and appearance of 

correspondence; (3) postage supplied; and (4) attitudinal and behavioral 
modifications (not discussed in this report) . Research indicates that 
multiple contacts are the most effective way to increase response rates, and 
that the SASS already includes most multiple contact techniques, with the 
exception of the use of special postage. Use of a fifth mail contact with 
two-day priority delivery is recommended. Another successful technique has 
been the inclusion of monetary incentives, a technique that does not seem 
appropriate for the SASS. The use of nonmonetary incentives, such as 
thank-you cards, is recommended. Making the questionnaires briefer and easier 
to complete is among additional recommendations. (Contains 7 tables and 24 
references . ) (SLD) 
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IMPROVING THE MAIL RETURN RATES OF SASS SURVEYS: 
A REVIEW OF THE LITERATURE 



INTRODUCTION 

Mail has been the data collection mode of choice of many who conduct surveys for many 
reasons. Some of the reasons noted by Kanuk and Berenson (1975) and by Mangione 
(1995) are: 

• Collecting data by mail is relatively inexpensive. There is no need to limit the 
sampling frame by geography because of the expense of reaching respondents in 
remote or distant areas 

• Mail surveys are easy to implement 

• There is no interviewer error since interviewers aren’t used 

• Mail questionnaire recipients have time to verify the validity of a survey if they 
choose to 

• Mail respondents have privacy, the flexibility of completing the questionnaire at 
their leisure, and time to make sure they are providing accurate information 

Although mail questionnaires are popular, response rates from mail returns alone are 
often quite low. Increasing mail survey response has motivated much of the research on 
mail surveys (Fox et al., 1988). In a 1991 article, Dillman notes that a “bibliography, 
including only items published since 1970, included more than 400 entries,” with the vast 
majority of mail survey research literature focusing on ways to increase the response rate. 

This paper presents the results of our review of mail survey literature and provides 
recommendations for the application of certain response-enhancing techniques to the 
Schools and Staffing Survey (SASS). 

SUMMARY 

Most of the research on improving mail survey response rates are of one of the following 
three types: 

• Single studies that evaluate the effect of one response-inducing technique 

• Comprehensive reports, based on results from previously published data, which 
evaluate the effectiveness of several response-inducing techniques 
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• Total design systems comprising lists of elements believed to be essential in 
obtaining optimal response rates 

We focused our attention on reports of the second type, with the intent of identifying 
response-inducing factors which have been shown to increase response rates across 
studies presented over several years. We found that incentives, multiple contacts, and 
respondent-friendly questionnaires are the response-enhancing techniques that have been 
shown to increase mail response rates across research studies and over time. 

BACKGROUND 

The Response Rate 

As illustrated below, the response rate of a survey is the proportion of the survey’s 
eligible sample units that completed questionnaires. The response rate tells what percent 
of the sample a researcher has actually measured 



„ „ ^ Interviewed sample units 

Response Rate = 

Eligible sample units 



Eligible cases = Interviews + Non-interviews 



Since the responses from a sample survey’s questionnaire are used to estimate population 
parameters, the degree to which sample estimates truly represent population parameters 
depends upon how similar the survey’s respondents and nonrespondents are. As the 
response rate of a survey increases, errors in the estimates due to nonresponse decrease. 
The higher the response rate, the more accurate the survey (Kanuk and Berenson, 1975). 

The Schools and Staffing Survey (SASS) 

SASS is an integrated set of surveys sponsored by the National Center for Education 
Statistics (NCES) and conducted by the Bureau of the Census. SASS provides data on 
public and private schools, teachers, students, libraries, librarians, principals, and public 
school districts. These data are used by educators, researchers, and policy makers. 



SASS components are: 



• The Teacher Demand and Shortage Survey 

• The School Administrators Survey 

• The School Survey 

• The Teacher Survey 

• The Library Survey 

• The Librarian Survey 

• The Student Records Survey 

• The Teacher Follow-up Survey (TFS) 

Another survey, which is not a part of SASS but is sponsored by NCES and conducted by 
the Census Bureau is the Private School Survey (PSS). Our discussion is also applicable 
to this survey. 

SASS uses mail questionnaires as its primary method for data collection. The survey 
response rates and the initial and second mail return rates from the most recent (1994) 
SASS are listed below in table 1 . As seen in table 1, only between 25 and 75 percent of 
the questionnaires had been returned after the second mailing of questionnaires. It would 
not have been wise to produce SASS estimates based on the responses from mail returns 
alone. 

After the second mailing of questionnaires, the more expensive, follow-up phase of data 
collection began, with centralized and decentralized telephone interviewing and a minute 
number of personal visit (face-to-face) interviews. The first column of table 1 shows 
unweighted final response rates for the 1994 SASS were between about 80 and 100 
percent, a considerable increase from the much lower mail return rates. An increase in the 
mail return rates would reduce the number and expense of telephone and face-to-face 
interviews, reducing the overall cost of interviewing for SASS. 



M 



Table 1. Unweighted Questionnaire Response and Mail Return Rates 



Survey Type 


Response Rate^ 


Initial Mail 
Return Rate^ 


Final Mail 
Return Rate^ 


Teacher Listing Form: Public Schools 


95.0 


40.6 


52.7 


Teacher Listing Form: Private Schools 


91.8 


36.9 


48.8 


Teacher Demand and Shortage (LEA) 


93.1 


51.3 


56.9 


Public School Administrator 


96.6 


52.9 


69.3 


Private School Administrator 


90.3 


47.2 


59.0 


Indian School Administrator 


98.7 


16.3 


47.5 


Public School 


92.0 


39.7 


55.6 


Private School 


84.1 


39.4 


52.3 


Indian School 


99.3 


37.5 


53.8 


Public Teacher^ 


88.9 


16.9 


31.9 


Private Teacher^ 


80.6 


11.1 


24.5 


Indian Teacher 


87.1 


25.1 


33.4 


Student Survey^ 


91.1 


32.5 


68.4 


Public School Library 


93.5 


44.2 


61.4 


Private School Library 


77.7 


33.5 


56.7 


Indian School Library 


83.9 


40.6 


64.4 


Public School Librarian 


89.4 


49.3 


72.5 


Private School Librarian 


88.3 


36.4 


60.4 


Indian School Librarian 


90.2 


44.4 


68.1 


Private School Survey (PSS) 


99.0 


45.7 


66.6 


Teacher Follow-Up Survey (TFS) 


88.2 


27.9 


39.1 


Stayers 


88.6 


33.7 


44.1 


Leavers 

— ■ _ r ■ . VT . . — ^ r- 


87.5 


19.1 


28.2 



^These columns include all cases that are final and do not need telephone follow-up or a second mailing. That is, they include 
interviews, noninterviews, out-of-scope cases. (Source: 1993-94 SASS Data Collection Progress Reports.) 

^These rates do not include the 5 percent of public schools that did not provide teacher lists. 

^These rates do not include the 9 percent of private schools that did not provide teacher lists. 

^These rates do not include the 12 percent of public schools, the 2 1 percent of private schools, and the 6 percent of Indian schools that 



did not participate in student sampling. 
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Sources: Teacher Listing Form: U.S. Department of Commerce, Bureau of the Census (1994), J994 Teacher Listing Form Check-in 

Report, Teacher Follow-up Survey: Whitener, Gruber, Colaciello et al. (1997, forthcoming), 1994-95 Teacher Followup 
Survey Data File User's Manual, Private School Survey: U.S. Department of Commerce, Bureau of the Census (1996), 
1996 PSS Check In Report, All others: Abramson et al. (1996), 1993-94 Schools and Staffing Survey: Sample Design and 
Estimation (NCES 96-089). 
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The data collection schedules for the NCES surveys are in tables 2A, 2B, and 2C below. 



Table 2A. 1993-94 SASS Data Collection Time Schedule 



Activity 


Date 




Introductor}^ letters mailed to local education agencies (LEAs) 


Aug. 1993 




Introductory letters and teacher listing forms mailed to schools 


Sept. 1993 




Census field representatives called school districts to obtain the name of the 
contact person to whom the Teacher Demand and Shortage 
Questionnaire should be addressed 


Sept. 1993 




Second mailing of teacher listing forms to schools 


Oct. 1993 




First mailing of questionnaires to LEAs and of school principal, library and 
librarian questionnaires to schools 


Oct. 1993 




Telephone follow-up of teacher listing forms not returned by school 


Nov. - Dec. 


1993 


Second mailing of LEA, principal, library and librarian questioimaires 


Nov. - Dec. 


1993 


First mailing of school questioimaires 


Dec. 1993 




First mailing of teacher questioimaires to schools 


Dec. 1993 - 


Feb. 1994 


Advance letters mailed to schools selected for the student records survey 


Dec. 1993 




Telephone calls to schools for the student records survey sample selection 


Jan. -Feb. 1994 


Second mailing of school and teacher questioimaires 


Jan. -Feb. 1994 


First mailing of student questioimaires to schools 


Mar. 1994 




Second mailing of student questioimaires to schools 


Apr. 1994 




Personal visit sample selection and interviews for student records survey 


Mar. - June 1994 


Telephone follow-up of mail nonrespondents 


Jan. - June 1994 



Source: Gruber et al. (1996). 1993-94 Schools and Staffing Survey: Data File User's hdanual (NCES 96-142). 
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Table 2B. 1995 TFS Data Collection Time Schedule 



Activity 


Date 


Advance letters mailed to LEAs and state administrators 


Aug. 1994 


Teacher Status Forms (TFS-1) and letters mailed to sample schools 


Sept. 1994 


Reminder postcards mailed to sample schools 


Sept. 1994 


Telephone follow-up of Teacher Status Forms not returned by 
schools 


Oct. - Nov. 1994 


Initial mailing of leaver/stayer questionnaires (TFS-2 and TFS-3) 


Jan. 1995 


Second mailing of leaver/stayer questionnaires (TFS-2 and TFS-3) 


Feb. 1995 


Telephone follow-up of mail questionnaire nonrespondents 


Mar. - May 1995 


Source: Whitener et al. (1997). Characteristics of Stayers, Movers, and Leavers: Results from the Teacher 
Followup Survey: 1994-95 (NCES 97-450). 


Table 2C. 1996 PSS Data Collection Time Schedule 




Activity 


Date 


First mailing of questionnaires to schools 


Oct. 1995 


Reminder postcards one week after first mailing of questionnaires 
to schools 


Oct. 1995 


Second mailing of questionnaires 


Nov. 1995 


Reminder postcards one week after second mailing of 
questionnaires 


Nov. 1995 


CATI foUow-up 


Feb. - May 1996 


Personal visit foUow-up 


June - July 1996 



Source: Broughman and Colaciello (1997, forthcoming). Private School Universe Survey 1995-1996 
(NCES 97-458). 
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TECHNIQUES FOR IMPROVING MAIL SURVEY RESPONSE RATES 



The response rate enhancement techniques that have been manipulated over the years can 
be grouped into the following four general categories: 

Motivating a response 

• Cash and non-cash incentives 

• Multiple follow-ups (reminder post cards, additional questionnaire mailings, and 

telephone follow-ups, timing and number of follow-ups) 

• Promise of anonymity or confidentiality 

• Prenotification 

Content and appearance of correspondence 

• Survey sponsorship 

• Personalization of correspondence 

• Content of the cover letter 

• Questionnaire wording, layout, color, length, and topic 
Postage 

• Type of outgoing and return postage (whether stamped or franked) 

• Rate of postage (the use of special delivery) 

The fourth group of techniques, based on attitudinal and behavioral theories of sociology 
and psychology, was found in more recent literature. A list of some of these ideas* 
follows. We will not discuss them further in this report. 

Attitudinal and Behavioral 

• Developing respondent burden indexes and finding ways to decrease the burden 

• Using cognitive research to develop respondent-friendly questionnaires 

• Conducting attitude studies to find out why people participate in surveys and use 

the results to increase participation 

• Targeting certain subpopulations known to have lower response rates 

• Pretesting cover letters and questionnaires using focus groups and other such 

methods 

• Incorporating the idea that a limit to the response rate for a survey is determined 

by elements like the survey’s subject matter and population of interest. 
This “threshold” response rate should be determined and accepted. 



* 



Lyberg, Lars and Patricia Dean, “Methods for Reducing Nonresponse Rates: A Review”, a paper presented at the 1992 
Annual Meeting of the American Association for Public Research. 



There are numerous reports on studies that were conducted to improve mail survey 
response rates. The majority of the studies measure the effect of manipulating one factor 
for a specific survey. Oftentimes, the results from studies involving the same factor are 
inconsistent from study to study. There are also many comprehensive reports where the 
authors have combined results from previously published reports and analyzed the 
effectiveness of more than one factor across several studies. 

Instead of perusing numerous reports from single-element studies to extract universal 
conclusions about the effectiveness of various techniques, we elected to study several of 
the comprehensive reports, since the authors of these studies have already performed 
much of this task for us. Also, because of technological changes and changes in society 
and its perceptions, the effect of a particular technique on increasing response rates to 
mail surveys could change over time. We looked at reports that were presented across the 
last two decades, with hopes of finding the techniques that consistently led to increased 
mail response rates, not only across studies, but also over time. 

The reports are discussed in Section V below. 



COMPREHENSIVE REPORTS ON IMPROVING RESPONSE TO MAIL 
SURVEYS 



Mail Survey and Response Rates: A Literature Review 

Kanuk and Berensen’s 1975 review of empirical studies to increase response rates for 
mail questionnaire surveys provides a look at the state of mail questionnaire research up 
to the time. The authors evaluated several techniques, one technique at a time, and made 
an assessment of the success of the techniques across studies. A summary of what they 
found is provided below in table 3. 

Table 3. Effects of Techniques on Response Rates 



Technique 


Conclusion 


Advance Notification* 


Effective in increasing the rate of return, but not better than 
foUow-up mailings 


FoUow-up Techniques* 


Successful across experiments; each successive contact resulted 
increased returns; the required cost of the additional follow-up 
should be weighed against the value of the additional information 
obtained 


Questionnaire Length 


Short questionnaires are not necessarily more likely to result in 
higher response rates than longer questionnaires 


Survey Sponsorship* 


Official or university sponsors have higher returns over 
commercial sponsors 


Return Envelopes* 


Shown to increase return, but only one study was found which 
tested the technique 


Postage — Outgoing and Return 


Inconsistent results 


Personalization 


Inconclusive results 


Cover Letter 


Produced no significant differences in response rates 


Anonymity 


Little evidence to support the assumption that promises of 
anonymity or confidentiality led to improved response rates 


Size, Reproduction, and Color of 
Questionnaires 


Produced no significant differences in response rates 


Premiums and Rewards* 


Very effective in increasing response rates. For middle-class 
respondents, promised rewards produce very little increases in 
response, whereas immediate rewards are more effective; the 
opposite is true for poorer respondents. 



* Led to improved response 

Source: Kanuk and Berensen (1975). Mail surveys and response rates: A literature review. 



Factors Affecting Response Rates to Mailed Questionnaires: A Quantitative Analysis 
of the Published Literature 

Heberlein and Baumgartner (1978) wanted to conduct a factorial experiment to study the 
results of simultaneously varying all the factors they believed had an influence on the 
return of mail questionnaires. They realized that such a venture would be an enormous 
undertaking, then settled for what they considered a “second best” approach. 

They examined the effect of several factors on an average response rate using results from 
already published studies and techniques from regression analysis. They treated the 
published results from 98 experiments as if they were respondents in a survey, coded the 
data from the published reports, and identified 71 factors to determine their effect on 
response rates. 

Their study led to the following: 

• An average response rate of 48 percent after the first mailing (with a range of 20% 
to 80%) 

• A final response rate of 61 percent with over one-fourth of the studies having 
more than 80 percent final response 

• A regression equation with the 10 variables listed in table 4 were identified as 
predictors of the final response rate 

• A causal model of response rates which explained 98 percent of the variance in 
response rates across studies. The number of contacts and the salience of the 
survey to the respondent explained 51 percent of the variance in the final 
response 



Table 4. 



Results for Factors Affecting Response Rates 



Factor 



Results 



Sponsorship 



Population 



1 ) Market Research 
Background 

2) Government 
Organization* 



Unlike with marketing agencies, respondents feel that 
their input to government-sponsored surveys may 
result in changes in public policy that they could see 



3) General Population 

4) Employee Population 

5) School or Army 
populations 



Surveys that involve the general population have lower 
response rates than those that involve certain 
subgroups of the population — such as employees or 
military personnel. 



Questionnaire 6) Saliency of the topic* 



7) Length-Number of 
pages 



If a respondent is knowledgeable and/or interested in 
the topic, he will probably also be more confident that 
his personal input will be of some importance to the 
study. 

Response rates decrease proportionally with the 
increase in the number of pages in the questionnaire. 



FoUow-up 



Incentives 



The author suggests reducing the number of pages in 
follow-up questionnaires by including only the most 
essential data, reasoning that the shorter questionnaire 
may make the recipient more willing to complete it 
since the burden is reduced. 



8 ) 

9) 



Total number of If a questionnaire is not returned after the second 

contacts* mailing, the third attempt to contact respondents 

Special Third Contact* should be special attempts, like a telephone call or 
special delivery mailing. The special attention and 
effort of additional mailings or special contacts would 
illustrate the energy the researcher is willing to expend 
to get the questionnaire recipients' input, making 
respondents feel that their response is indeed 
important. 



10) Monetary Incentives 
with the First Contact* 



(*) Indicates a positive influence on response 



Mail Survey Response Rates: A Meta-Analysis of Selected Techniques for Inducing 
Response 

Fox and others (1988) performed a meta-analysis using results from published studies that 
investigated the individual influence of response rate enhancement techniques on the 
response rate of surveys conducted by mail. Their analysis evaluated the following 
techniques; 

• Prenotification by letter 

• Follow-up by postcard 

• Outgoing postage (first-class versus bulk rate & stamped versus metered) 

• Stamped return postage versus business reply 

• Notification of the closing date (for return of questionnaires) 

• University sponsorship versus business sponsorship 

• Color of the questionnaire (green versus white) 

.• Handwritten note asking for cooperation 

• Monetary incentives 

In meta-analysis, the results from various published studies are combined and used to 
compute an overall or average value. The average or overall value is then tested for 
statistical significance. For their analysis, Fox and his colleagues used the nine individual 
response rate factors above to compute an aggregate estimate of effect size, where the 
effect was measured as the difference between the response rates of a control and a 
treatment group. They determined whether the aggregate estimate was statistically 
significant or not by accumulating evidence against the null hypothesis of “no effect” 
across the studies. 

In their study, they did the following: 

• They created a control group and assigned a “zero” level of treatment 

• They performed individual tests of the hypothesis of “no effect” on the response 
rate from each study (z-tests) and combined individual test results using the chi- 
square statistic to determine an overall level of significance. 

• To determine the average response rate effect size, they tested the hypothesis that 
“the effect of the aggregate estimate is zero” using z-tests, where the z-value used 



was the ratio of a minimum variance estimate of the response rate effect to the 
estimate of its standard deviation. 

Based on the average response rate effect sizes, the authors concluded that university 
sponsorship, prenotification letters, postcard follow-ups, first-class outgoing postage, 
stamped return postage, and the color of the questionnaire, all led to increases in the 
response rate, with university sponsorship producing the largest increase in the response 
rate. 

Influence of 13 Design Factors on Completion Rates to the Decennial Census 
Questionnaires 

This report (1994) combines results from four nationwide studies which were performed 
to assess the effectiveness of several factors on the rate at which U. S. Decennial Census 
questionnaires are returned. The results of this research are from tests of techniques to 
increase the mail return rates of a census, which is mandatory by law, rather than to a 
survey, which is strictly voluntary. However, we believe most of its findings can be 
applied to mail surveys in general. 

This report evaluates the combined results from four National Census Tests to make 
recommendations about their application to the 2000 Decennial Census. The report also 
compares completion rates between treatments nationally, and between the low and high 
1990 Census response rate strata. The four tests were: 

• The Simplified Questionnaire Test (SQT), in which respondent friendly 
questionnaire design, the inclusion of a difficult question, questionnaire length, 
and the use of replacement questionnaires were tested. 

• The Implementation test (IT), in which the use of prenotice letters, stamped return 
envelopes, and reminder postcards were tested. 

• The Mail and Telephone Mode Test (MTMT), in which an invitation to reply by 
telephone, follow-up letters with telephone invitation, different numbers of 
contacts, and replacement questionnaires on response to short respondent-friendly 
questionnaires were tested. 

• The Appeals and Long-Form test (ALFE), in which two kinds of respondent- 
friendly construction on responses to long forms, benefits appeal versus 
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mandatory appeal, and alternative confidentiality statements on response to short, 
respondent-friendly questionnaires were tested. 

Together the four experiments included 27 treatment panels and tested the effect of the 
following 13 variables on completion rates: 



A request for social security number (SSN) 

A stamped return envelope (ST) 

A prenotice letter (PN) 

A reminder postcard (REM) 

A replacement questionnaire mailing (RQ) 

A follow-up letter after the reminder postcard (FUL) 

An invitation to respond by telephone (TI) 

Respondent-friendly versus, traditional questionnaires (RF) 

Four variations of questionnaire length (Short, Shorter, Shortest, Long) 
A strong (C"^ ) versus standard confidentiality statements 
A benefit appeal on the envelope and a motivational insert (BEN) 

A mandatory appeal (M) 

Inserts with either a mandatory or benefits appeal (I) 



The universe for each test was the same. It consisted of 88.8 million housing units from 
1990 decennial census “mail-back” areas, which are geographic regions of the country 
with good addresses. They excluded any unit which had been selected for a previous test 
and units for which the U. S. Postal Service would probably not deliver mail. 

Findings from the study are reproduced in tables 6 and 7. In the contribution column, 
positive values signify the factor added significantly to the results and negative values 
signify the factor subtracted significantly from the results. Table 7 shows a consistent 
disparity between the low and high response strata. Summarizing the results shown in 
table 6, 

Six factors had little or no effect on the return of questionnaires 

• Stamped return envelopes 

• Inviting people to call in their answers by telephone 

• A benefit appeal on the envelope and cover of an insert 

• A stronger confidentiality message in the follow-up letter 
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• A stronger confidentiality message 

• A follow-up letter with a replacement questionnaire 

Two factors had negative effects on the return of questionnaires 

• Request for social security number 

• Length of questionnaires 

Five factors had positive effects on the return of questionnaires 

• The prenotice letter 

• Reminder postcards 

• Replacement questionnaire mailing 

• Respondent-friendly questionnaire design 

• A printed message stating that answering the Census is mandatory by law 

The authors of the study concluded that no single factor would guarantee an increase in 
response to the 2000 Census. They believe a viable plan would be to jointly use the five 
factors which consistently showed improvements across the four experiments. 
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Table 5. 



National Completion Rates and Contributions of the Design Factors 
in the Presence of Selected Variables 



Factor 


In the Presence of 


Contribution 


Comparison 


SSN 


Shorter, RF, PN, REM, RQ 


-3.4* 


SQT(4) - SQT(3) 


ST 


Shorter, RF 


2.5 


IT(3) - ITG) 




Shorter, RF, PN 


3.4* 


IT(5) - IT(2) 




Shorter, RF, REM 


1.5 


IT(6) - IT(4) 




Shorter, RT, PN, REM 


1.6 


IT(8) - IT(7) 


PN 


Shorter, RF 


6.4* 


IT(2) - ITG) 




Shorter, RF, ST 


7.3* 


IT(5) - IT(3) 




Shorter, RF, REM 


4.7* 


IT(7) - IT(4) 




Shorter, RF, ST, REM 


4.8* 


IT(8) - IT(6) 


REM 


Shorter, RF 


8.0* 


IT(4) - ITG) 




Shorter, RF, PN 


6.3* 


IT(7) - IT(2) 




Shorter, RF, ST 


6.9* 


IT(6) - IT(3) 




Shorter, RF, PN, ST 


4.4* 


IT(8) - IT(5) 


RQ 


Shorter, RF, PN, REM 


10.5* 


SQT(3) - IT(7) 




Short, RF, PN, REM, TI 


7.9* 


MTMT(l) - MTMT(2) 




Short, RF, PN, REM, FUL, TI 


6.1* 


MTMT(4) - MTMT(3) 


FUL 


Short, RF, PN, REM, TI 


3.2* 


MTMT(3) - MTMT(2) 




Short, RF, PN, REM, RQ, IT 


1.5 


MTMT(4) - MTMT(l) 


PN&REM 


Shorter, RF 


12.7* 


IT(7) - ITG) 




Shorter, RF, ST 


11.7* 


IT(8) - IT(3) 


PN&RQ 


Shorter, RF, REM 


15.0* 


SQT(3) - IT(4) 


REM&RQ 


Shorter, RF, PN 


16.7* 


SQT(3) - IT(2) 


PN,REM&RQ 


Shorter, RF 


22.5* 


SQT(3) - ITG) 


FUL&RQ 


Short, RF, PN, REM, TI 


9.4* 


MTMT(4) - MTMT(2) 


TI 


Short, RF, PN, REM, RQ 


-1.4 


MTMT(5) - MTMT(l) 


RF 


Short, IS, PN, REM, RQ 


3.4* 


SQT(2)-SQT(1) 




Short, IS, PN, REM, RQ, TI 


1.6 


MTMT(5) - SQTG) 




Long, IS, PN, REM, RQ 


4.1* 


ALFE(2) - ALFE(l) 




Long, RC, PN, REM, RQ 


2.6 


ALFE(3)-ALFE(1) 


Short 


Long, RF, PN, REM, RQ 


11.3* 


ALFE(4) - ALFE(2) 




Long, PN, REM, RQ 


11.6* 


SQT(1)-ALFE(1) 


Shorter 


Short, RF, PN, REM, RQ 


4.6* 


SQT(3) - SQT(2) 




Long, RF, IS, PN, REM, RQ 


15.4* 


SQT(3) - ALFE(2) 
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Table 5. National Completion Rates and Contributions of the Design Factors 
in the Presence of Selected Variables, Continued 



Factor 


In the Presence of 


Contribution 


Comparison 


Shortest 


Short, RF, PN, REM, RQ 


4.1* 


SQT(5) - SQT(2) 




Shorter, RF, PN, REM, RQ 


-0.4 


SQT(5) - SQT(3) 




Long, RF, IS, PN, REM, RQ 


15.0* 


SQT(5) - ALFE(2) 




Long, RF, RC, PN, REM, RQ 


16.5* 


SQT(5) - ALFE(3) 


Shorter & RF 


Short, PN, REM, RQ 


8.0* 


SQT(3) - SQT(L) 




Long, PN, REM, RQ 


19.6* 


SQT(3)-^F£r/; 


Shortest & RF 


Short, PN, REM, RQ 


7.5* 


SQT(5) - SQTG) 




Long, PN, REM, RQ 


19.1* 


SQT(5) - ALFE(l) 


Long, IS & RF 


Short, PN, REM, RQ 


-7.5* 


ALFE(2) - SQT(l) 


Long, RC & RF 


Short, PN, REM, RQ 


-9.0* 


ALFE(3) - SQT(l) 


C+ 


Short, RF, PN, REM, RQ, BEN 


0.6 


ALFE(5) - ALFE(6) 




Short, RF, PN, REM, RQ, MAN 


-1.1 


ALFE(7) - ALFE(8) 


BEN 


Short, RF, PN, REM, RQ, C+, I 


1.8 


ALFE(5) - ALFE(4) 




Short, RF, PN, REM, RQ, C, I 


1.2 


ALFE(6) - ALFE(4) 


MAN 


Short, RF, PN, REM, RQ, C+, I 


9.8* 


ALFE(7)-ALFE(4) 




Short, RF, PN, REM, RQ, C, I 


10.9* 


ALFE(8) - ALFE(4) 




Short, RF, PN, REM, RQ 


9.1* 


ALFE(9) - ALFE(4) 


I 


Short, RF, PN, REM, RQ, MAN, C+ 


0.7 


ALFE(7) - ALFE(9) 




Short, RF, PN, REM, RQ, MAN, C 


1.7 


ALFE(8) - ALFE(9) 



* Indicates that the difference is statistically significant at a = 0.10 



Source: U.S. Department of Commerce, Bureau of the Census, Decennial Statistical Support Division. 2000 Census 
Memorandum Series #E-85. 

SQT 
IT 

MTMT 
ALFE 
SSN 
ST 
PN 
REM 
RQ 
FUL 
TI 
RF 

Short, Shorter, Shortest, Long 
C",C 
BEN 
M 
I 



Simplified Questionnaire Test 
Implementation Test 
Mail and Telephone Mode Test 
Appeals and Long-Form Test 
Social Security Number 
Stamped return envelope 
Prenotice letter 
Reminder postcard 
Replacement questionnaire 
Follow-i]^) letter 

Invitation to respond by telephone 
Respondent-friendly versus traditional questionnaire 
Questionnaire length 

Strong versus standard coniidentiality statement 
Benefits appeal on the envelope 
Mandatory appeal 

Insert with either a mandatory or benefits appeal 



Table 6. 



Completion Rates for Treatments in Each Test 

Completion Rates 



Test (Treatment) 


Components 


National 


High Response 
Areas (HRA) 


Low Response 
Areas (LRA) 


SQT(l) 


Short, PN, REM, RQ 


63.4 


65.8 


45.2 


SQT(2) 


Short, RF, PN, REM, RQ 


66.8 


68.7 


52.7 


SQT(3) 


Shorter, RF, PN, REM, RQ 


71.4 


73.5 


55.1 


SQT(4) 


Shorter, RF, PN, REM, RQ, SSN 


68.0 


70.5 


48.9 


SQT(5) 


Shortest, RF, PN, REM, RQ 


70.9 


73.1 


54.6 


IT(1) 


Shorter, RF 


50.0 


51.9 


36.3 


IT(2) 


Shorter, RF, PN 


56.4 


58.6 


40.5 


IT(3) 


Shorter, RF, ST 


52.6 


54.5 


37.9 


IT(4) 


Shorter, RF, REM 


58.0 


60.2 


42.0 


IT(5) 


Shorter, RF, PN, ST 


59.8 


62.1 


43.0 


IT(6) 


Shorter, RF, ST, REM 


59.5 


61.8 


42.6 


IT(7) 


Shorter, RF, PN, REM 


62.7 


65.0 


45.4 


IT(8) 


Shorter, RF, PN, ST, REM 


64.3 


66.5 


47.8 


MTMT(1) 


Same as SOT(2) 


70.6 


72.7 


54.9 


MTMT(2) 


Short, RF, PN, REM, TI 


62.7 


65.1 


44.7 


MTMT(3) 


Short, RF, PN, REM, FUL, TI 


66.0 


68.4 


48.1 


MTMT(4) 


Short, RF, PN, REM, FUL, RQ, TI 


72.2 


74.3 


54.9 


MTMT(5) 


Short, RF, PN, REM, RQ, TI 


69.3 


71.5 


52.5 


ALFE(l) 


Long, PN, REM, RQ 


51.8 


53.6 


37.8 


ALFE(2) 


Long, RF, IS, PN, REM, RQ 


55.9 


58.4 


37.0 


ALFE(3) 


Long, RF, RC, PN, REM, RQ 


54.4 


56.4 


39.0 


ALFE(4) 


Same as SQT(2) 


67.2 


69.2 


52.3 


ALFE(5) 


Short, RF, PN, REM, RQ, BEN, C+, I 


69.1 


71.5 


50.5 


ALFE(6) 


Short, RF, PN, REM, TcQ, BEN, C, I 


68.4 


70.7 


51.5 


ALFE(7) 


Short, RF, PN, REM, RQ, MAN, C+, I 


77.0 


79.3 


59.7 


ALFE(8) 


Short, RF, PN, REM, RQ, MAN, C, I 


78.1 


80.5 


59.7 


ALFE(9) 


Short, RF, PN, REM, RQ, MAN 


76.4 


78.5 


60.7 



* Source: U.S. Department of Commerce, Bureau of the Census, Decennial Statistical Si^port Division. 2000 Census 
Memorandum Series #E-85. Formulae are reported in Sinclair and West, 1992; Sinclair et al., 1993: West, 1993: and 
Treat, 1993a, 1993b. 





Self Administered/Mail Surveys: A Two-day Short Course 



In the course material from his two-day short course on Mail Surveys (Universities of 
Maryland/Michigan Short Course, February 21-22,1996), Dillman discussed his Total 
Design Method (TDM), a comprehensive system for improving the response rates of mail 
surveys. With TDM, the objective is to design the elements in the survey process to fit 
together such that optimum response is achieved for the survey. The aim of TDM is a 
survey design where the prospective respondents view the rewards of responding to the 
survey as outweighing the costs. 

The TDM includes a list of elements (below) which are believed to be essential to 
obtaining good response rates. The list was provided in a summary presented in the two- 
day course. 



Table 7. Elements of the TDM Design 

Booklet questionnaires - 2, 6, 10, 14 page units 

Interesting cover/neutral 

Question order - least to most ‘"costly” 

Connect first question to letter^teresting 
Consistency of detail 
Vertical flow 

Instructions where needed, not in advance 

One request at a time 

Provide clear respondent paths 

Don't use same answer format for different tasks 

Type size/darkness to separate Q’s from answers 

Reduced type OK, usually 

Avoid cross-match and other "hard" formats 

Keep prose simple 

Transitional statements 

Pretest 



Source: Dillman (1996). Self-Administered/Mail Surveys. 



In the 1996 course material, Dillman says that adhering to TDM had significant impact 
on response rates. The more the survey adhered to the elements of TDM the higher their 
response rates. He also provided a critique of his TDM. He says the major weakness of 
TDM is the system’s “one fits all approach”, which does not allow for differences 
between surveys. In a 1991 article on the design and administration of mail surveys, he 
says it may not be necessary to include all details of TDM. 

As an update to his TDM, which he originally introduced in 1978, Dillman suggests the 
following: 

• Four contacts by mail -- first class; e.g., prenotice, questionnaires, reminders. 
Replacement questionnaires 

• A fifth mail contact by two-day priority mail 

• A financial incentive that is modest and prepaid 

• A contact by telephone, if possible 

• Personalized correspondence 

• Respondent-friendly questionnaire design that considers length, layout, and 
salience 

• Targeting specific populations 

Together, the elements comprise what he calls a “response maximization model” 
(Dillman, 1996). 



CONCLUSIONS 



Mail surveys are popular because of low costs and the ease of implementation. However, 
mail surveys are also known to have low mail return rates. If a survey’s response rate is 
too low, the effort and expense of designing and conducting the survey is wasted, since 
the accuracy and validity of the survey’s results are questionable and possibly unusable. 
Low mail return rates have motivated researchers over the years to search for ways to 
increase response to mail surveys. 

SASS, as do many other mail surveys, uses other data collection modes (like telephone 
and face-to-face interviewing) to follow-up mail nonrespondents. However, collecting 
data by these modes is more expensive than mail, and data collected in these modes are 
more prone to additional error since interviewers are used. An increase in SASS mail 
return rates would reduce the number of respondents that need to be reached by telephone 
or personal visit, thereby decreasing the overall cost of conducting a survey. 

Across studies and over time, multiple contacts, interesting and easy to follow 
questionnaires, and incentives have been consistently effective in improving mail survey 
response rates. 

Comments on Data Collection in SASS 

• In the 1994 SASS, prenotice letters were sent only to LEAs and schools. Other 
potential respondents were not notified in advance of the forthcoming 
questionnaire. 

• Multiple contacts were made: 

1) an initial questionnaire mailing 

2) a reminder postcard 

3) a second mailing of the questionnaire 

4) follow-up by telephone or personal visit 

• The major portion of the burden of completing SASS questionnaires rested on the 
shoulders of sample schools, since teacher listing forms and school and student 
questionnaires were completed by school staff or principals, and the principal 
questionnaire was completed by principals. 



Vn. RECOMMENDATIONS 



Follow-Ups 

Multiple contact techniques are the most effective techniques for increasing mail survey 
response rates. SASS data collection (table 3) involves prenotification, two mailings of 
questionnaires, reminder postcards, and nonresponse follow-ups by telephone or face-to 
face interviewing. 

The current SASS data collection procedure appears to already include most of the 
multiple contact techniques we found in the literature. The only technique associated with 
multiple respondent contact that has not been incorporated in SASS is the use of special 
postage. Reports from studies which used special delivery mail yielded inconsistent, but 
mostly positive results. We recommend the incorporation of Dillman’s (1996) 
suggestions of first class mail delivery of questionnaires and a fifth mail contact by two- 
day priority mail be considered for SASS. 

Incentives 

SASS response rates for 1994 generally decreased for each additional form we asked 
them to fill out (table 1). This decrease may be linked to response burden. School 
principals and/or his staff have the task of responding to multiple SASS questionnaires, 
since a principal and/or his staff complete the teacher listing form, administrator, school, 
and student record questionnaires. We recommend providing an incentive to respondent 
units who complete more than one SASS questionnaire. 

With the exception of additional follow-ups, the inclusion of monetary incentives with 
the questionnaire has been shown to be the most successful technique for improving mail 
survey response rates (Mangione, 1995). It’s probably not wise or permissible to offer 
financial incentives, to have SASS questionnaire completion rate contests by school 
districts, or to give away pins or bumper stickers that say “I’VE RETURNED MY SASS 
QUESTIONNAIRE,” so we recommend offering some non-monetary incentives to SASS 
respondents. Thank-you cards, offers to share summaries of survey results, or other 
incentives may be sufficient enticements to offset the burden of completing additional 
questionnaires. 



We recommend that the incentives be provided with the questionnaires, since results from 
meta-analysis research on incentives in mail surveys found that incentives provided with 
the initial mailing of the questionnaire had a positive effect on response rates, whereas 
incentives which were to be sent upon the return of the questionnaire did not (Church, 
1993). 

Questionnaires 

Respondents are more likely to complete and return questionnaires that are short, 
interesting, and easy to follow (Dillman et al., 1993 and 1994). We recommend 
scrutinizing the length of SASS questionnaires, the clarity of its instructions, and the 
format and sequence of its questions (Mangione, 1995) in an effort to ensure that all 
SASS questionnaires are respondent-friendly. In a profession already burdened with 
paperwork, short, interesting, and easy to follow questionnaires should be a welcome 
sight to “paper- weary” eyes. 

Additional Suggestions for Consideration 

Target SASS Sample Teachers 

SASS Teacher Surveys have lower mail return rates than the other SASS components 
(table 1). Teachers are not notified in advance that SASS questionnaires are coming. 
Prenotification letters sent to SASS sample teachers may positively influence mail return 
results. Another suggestion is to expend extra efforts to “target” a subpopulation of 
teachers who, after investigation of past survey results, are identified as “late response” or 
“nonresponse” teachers. 

Use Voice Mail, Facsimile, and Personal Computers as Alternatives to Mail 

We are now in the electronic age where most public and private schools have telephones 
and personal computers. An additional suggestion to consider as a possible way to 
increase response, is to give respondents the option of returning their questionnaires by 
facsimile machines, completing SASS questionnaires electronically -- by computer (disks 
by mail) or through some type of automated voice mail system. Voice mail and computer 
respondents could complete their questionnaires in privacy, without the intrusion of an 
interviewer. Although the initial implementation of this suggestion would be expensive, 
the use of these modes could quickly become cost effective. Issues of confidentiality and 
anonymity must be addressed before any of these methods are seriously considered. 
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Vra. LIMITATIONS 



• Some of the results and recommendations are based on findings obtained through 
our comparison of comprehensive studies involving the analysis of published 
reports. The reports themselves may not be comparable to each other because of 
methodological and other differences. 

• The conclusions and recommendations we’ve provided are qualitative, based on 
subjective analysis. 

• Additional contacts (prenotification, thank-you cards, etc.) may not be feasible 
because of time limitations (length of the school year) associated with the SASS 
data collection process. The addition of incentives may not be cost-effective. 
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